{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 第16章 主成分分析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 习题16.1\n",
    "&emsp;&emsp;对以下样本数据进行主成分分析：\n",
    "$$\n",
    "X = \\left[ \n",
    "\\begin{array}{lll}\n",
    "2 & 3 & 3 & 4 & 5 & 7 \\\\\n",
    "2 & 4 & 5 & 5 & 6 & 8\n",
    "\\end{array}\n",
    "\\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：**  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答思路：**\n",
    "\n",
    "1. 给出主成分分析算法\n",
    "2. 自编程基于`numpy`库实现主成分分析算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答步骤：**  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第1步：主成分分析算法**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;主成分分析方法主要有两种，可以通过相关矩阵的特征值分解或样本矩阵的奇异值分解进行。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 相关矩阵的特征值分解算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第16.2.2节的相关矩阵的特征值分解算法具体步骤：\n",
    "\n",
    "> （1）对观测数据按照如下公式进行规范化处理，得到规范化数据矩阵，仍以$X$表示\n",
    "> \n",
    "> $$\n",
    "x_{ij}^* = \\frac{x_{ij} - \\bar{x}_i}{\\sqrt{s_{ii}}}, \\ i=1,2,\\cdots,m; \\ j = 1,2,\\cdots,n\n",
    "$$\n",
    "> \n",
    "> 其中\n",
    "> $$\n",
    "\\begin{array}{ll}\n",
    "\\displaystyle \\bar{x}_i = \\frac{1}{n} \\sum_{j=1}^n x_{ij}, \\quad i = 1,2,\\cdots,m \\\\\n",
    "\\displaystyle s_{ii} = \\frac{1}{n-1} \\sum_{j=1}^n (x_{ij} - \\bar{x}_i)^2, \\quad i = 1,2,\\cdots,m\n",
    "\\end{array}\n",
    "$$\n",
    ">（2）依据规范化数据矩阵，计算样本相关矩阵$R$\n",
    "> \n",
    "> $$\n",
    "R = [r_{ij}]_{m \\times m} = \\frac{1}{n-1} X X^T\n",
    "$$\n",
    "> \n",
    "> 其中\n",
    "> \n",
    "> $$\n",
    "r_{ij} = \\frac{1}{n-1} \\sum_{l=1}^n x_{il}x_{lj}, \\quad i,j=1,2,\\cdots,m\n",
    "$$\n",
    "> \n",
    "> （3）求样本相关矩阵$R$的$k$个特征值和对应的$k$个单位特征向量。\n",
    "求解$R$的特征方程\n",
    "> \n",
    "> $$\n",
    "|R - \\lambda I| = 0\n",
    "$$\n",
    "> \n",
    "> 得$R$的$m$个特征值\n",
    "> \n",
    "> $$\n",
    "\\lambda_1 \\geqslant \\lambda_2 \\geqslant \\cdots \\geqslant \\lambda_m  \n",
    "$$\n",
    "> \n",
    "> 求方差贡献度$\\displaystyle \\sum_{i=1}^k \\eta_i$达到预定值的主成分个数$k$。  \n",
    "> 求前$k$个特征值对应的单位特征向量\n",
    "> \n",
    "> $$\n",
    "a_i = (a_{1i}, a_{2i}, \\cdots, a_{mi})^T, \\quad i=1,2,\\cdots,k\n",
    "$$\n",
    "> \n",
    "> （4）求$k$个样本主成分  \n",
    "> 以$k$个单位特征向量为系数进行线性变换，求出$k$个样本主成分\n",
    "> \n",
    "> $$\n",
    "y_i = a_i^T x, \\quad i=1,2,\\cdots,k\n",
    "$$\n",
    "> \n",
    "> （5）计算$k$个主成分$y_i$与原变量$x_i$的相关系数$\\rho(x_i,y_i)$，以及$k$个主成分对原变量$x_i$的贡献率$v_i$。  \n",
    "> \n",
    "> （6）计算$n$个样本的$k$个主成分值  \n",
    "> 将规范化样本数据带入$k$个主成分式，得到$n$个样本的主成分值。第$j$个样本$x_j = (x_{1j}, x_{2j}, \\cdots, x_{mj})^T$的第$i$个主成分值是\n",
    "> \n",
    "> $$\n",
    "y_{ij} = (a_{1i}, a_{2i}, \\cdots, a_{mi})(x_{1j}, x_{2j}, \\cdots, x_{mj})^T = \\sum_{l=1}^m a_{li} x_{lj} \\\\\n",
    "i = 1,2,\\cdots,m, \\quad j=1,2,\\cdots, n\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第16章的定义16.2给出了方差贡献率：\n",
    "\n",
    "> **定义16.2** 第$k$主成分$y_k$的方差贡献率定义为$y_k$的方差与所有方差之和的比，记作$\\eta_k$\n",
    "> \n",
    "> $$\n",
    "\\eta_k = \\frac{\\lambda_k}{\\displaystyle \\sum_{i=1}^m \\lambda_i} \\tag{16.30}\n",
    "$$\n",
    "> \n",
    "> $k$个主成分$y_1,y_2,\\cdots,y_k$的累计方差贡献率定义为$k$个方差之和与所有方差之和的比\n",
    "> \n",
    "> $$\n",
    "\\sum_{i=1}^k \\eta_i = \\frac{\\displaystyle \\sum_{i=1}^k \\lambda_i}{\\displaystyle \\sum_{i=1}^m \\lambda_i}  \\tag{16.31}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 样本矩阵的奇异值分解算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第16章的算法16.1主成分分析算法\n",
    "\n",
    "> **算法16.1（主成分分析算法）**  \n",
    "> 输入：$m \\times n$样本矩阵$X$，其每一行元素的均值为零；  \n",
    "> 输出：$k \\times n$样本主成分矩阵$Y$。  \n",
    "> 参数：主成分个数$k$  \n",
    "> （1）构造新的$n \\times m$矩阵\n",
    "> $$\n",
    "X' = \\frac{1}{\\sqrt{n - 1}} X^T\n",
    "$$\n",
    "> $X'$每一列的均值为零。  \n",
    "> （2）对矩阵$X'$进行截断奇异值分解，得到\n",
    "> $$\n",
    "X' = U \\Sigma V^T\n",
    "$$\n",
    "> 有$k$个奇异值、奇异向量。矩阵$V$的前$k$列构成$k$个样本主成分。  \n",
    "> （3）求$k \\times n$样本主成分矩阵\n",
    "> $$\n",
    "Y = V^T X\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第2步：自编程基于`numpy`库实现主成分分析算法**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "def pca_svd(X, k):\n",
    "    \"\"\"\n",
    "    样本矩阵的奇异值分解的主成分分析算法\n",
    "    :param X: 样本矩阵X\n",
    "    :param k: 主成分个数k\n",
    "    :return: 特征向量V，样本主成分矩阵Y\n",
    "    \"\"\"\n",
    "    n_samples = X.shape[1]\n",
    "\n",
    "    # 构造新的n×m矩阵\n",
    "    T = X.T / np.sqrt(n_samples - 1)\n",
    "\n",
    "    # 对矩阵T进行截断奇异值分解\n",
    "    U, S, V = np.linalg.svd(T)\n",
    "    V = V[:, :k]\n",
    "\n",
    "    # 求k×n的样本主成分矩阵\n",
    "    return V, np.dot(V.T, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正交矩阵V：\n",
      "[[ 0.707  0.707]\n",
      " [ 0.707 -0.707]]\n",
      "样本主成分矩阵Y：\n",
      "[[-2.028 -0.82  -0.433  0.     0.82   2.461]\n",
      " [ 0.296 -0.046 -0.433  0.     0.046  0.137]]\n"
     ]
    }
   ],
   "source": [
    "X = np.array([[2, 3, 3, 4, 5, 7],\n",
    "              [2, 4, 5, 5, 6, 8]])\n",
    "X = X.astype(\"float64\")\n",
    "\n",
    "# 规范化变量\n",
    "avg = np.average(X, axis=1)\n",
    "var = np.var(X, axis=1)\n",
    "for i in range(X.shape[0]):\n",
    "    X[i] = (X[i, :] - avg[i]) / np.sqrt(var[i])\n",
    "\n",
    "# 设置精度为3\n",
    "np.set_printoptions(precision=3, suppress=True)\n",
    "V, vnk = pca_svd(X, 2)\n",
    "\n",
    "print(\"正交矩阵V：\")\n",
    "print(V)\n",
    "print(\"样本主成分矩阵Y：\")\n",
    "print(vnk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 习题16.2\n",
    "&emsp;&emsp;证明样本协方差矩阵$S$是总体协方差矩阵$\\Sigma$的无偏估计。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答思路：**  \n",
    "\n",
    "1. 给出总体协方差矩阵$\\Sigma$\n",
    "2. 给出样本协方差矩阵$S$\n",
    "3. 给出无偏估计的定义\n",
    "4. 证明样本协方差矩阵$S$是总体协方差矩阵$\\Sigma$的无偏估计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答步骤：**   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第1步：总体协方差矩阵$\\Sigma$**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第16章的定理16.1：\n",
    "\n",
    "> **定理16.1** 设$x$是$m$维随机变量，$\\Sigma$是$x$的协方差矩阵，$\\Sigma$的特征值分别是$\\lambda_1 \\geqslant \\lambda_2 \\geqslant \\cdots \\geqslant \\lambda_m \\geqslant 0$，特征值对应的单位特征向量分别是$\\alpha_1, \\alpha_2, \\cdots, \\alpha_m$，则$x$的第$k$个主成分是\n",
    "> $$\n",
    "y_k = \\alpha_k^T x = \\alpha_{1k} x_1 + \\alpha_{2k} x_2 + \\cdots + \\alpha_{mk} x_m, \\quad k =1,2,\\cdots, m  \n",
    "$$\n",
    "> $x$的第$k$主成分的方差是\n",
    "> $$\n",
    "\\text{var}(y_k) = \\alpha_k^T \\Sigma \\alpha_k = \\lambda_k, \\quad k = 1, 2, \\cdots, m\n",
    "$$\n",
    "> 即协方差矩阵$\\Sigma$的第$k$个特征值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第16章的本章概要的总体主成分的性质：\n",
    "> 主成分$y$的协方差矩阵是对角矩阵\n",
    "> $$\n",
    "\\text{cov}(y) = \\Lambda = \\text{diag}(\\lambda_1, \\lambda_2, \\cdots, \\lambda_m)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第2步：样本协方差矩阵$S$**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第16.2.1的样本协方差矩阵定义：\n",
    "\n",
    "> &emsp;&emsp;给定样本矩阵$X$，可以估计样本均值，以及样本协方差。样本均值向量$\\bar{x}$为\n",
    "> $$\n",
    "\\bar{x} = \\frac{1}{n} \\sum_{j=1}^n x_j \\tag{16.40}\n",
    "$$\n",
    "> 样本协方差矩阵$S$为\n",
    "> $$\n",
    "\\begin{array}{l}\n",
    "S = [s_{ij}]_{m \\times m} \\\\\n",
    "\\displaystyle s_{ij} = \\frac{1}{n - 1} \\sum_{k=1}^n (x_{ik} - \\bar{x}_i)(x_{jk} - \\bar{x}_j), \\quad i,j=1,2,\\cdots,m \n",
    "\\end{array}  \\tag{16.41}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第3步：无偏估计的定义**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据《实用多元统计分析》（方开泰，1989年9月第一版）中第88~89页的无偏性：\n",
    "> 如果参数$\\theta$的某个估计$\\hat{\\theta}$满足$E(\\hat{\\theta})=\\theta$，则称$\\hat{\\theta}$是无偏的。······引理3.4还指出：\n",
    "> \n",
    "> $$\n",
    "\\hat{\\Sigma} = \\frac{1}{n} A^d = \\frac{1}{n}\\sum_{\\alpha=1}^{n-1}z_{\\alpha} z'_{\\alpha}\n",
    "$$\n",
    "> \n",
    "> $z_1, \\cdots, z_{n-1}$独立同分布于$N_p(0, \\Sigma)$，于是\n",
    "> \n",
    "> $$\n",
    "E(\\hat{\\Sigma}) = \\frac{1}{n}\\sum_{\\alpha=1}^{n-1}z_{\\alpha} z'_{\\alpha} = \\frac{n-1}{n} \\Sigma\n",
    "$$\n",
    "> \n",
    "> $\\hat{\\Sigma}$不是$\\Sigma$的无偏估计，但可修正一下使之为无偏估计。令\n",
    "> \n",
    "> $$\n",
    "S = \\frac{1}{n-1}A\n",
    "$$\n",
    "> \n",
    "> 则$S$是$\\Sigma$的无偏估计。在实用中普遍采用$S$来估计$\\Sigma$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;无偏估计是用样本统计量来估计总体参数的一种无偏推断。估计量的期望等于被估计参数的真实值，则称此估计量为被估计参数的无偏估计，即具有无偏性。表示在多次重复下，它们的平均数接近所估计的参数真值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第4步：证明样本协方差矩阵$S$是总体协方差矩阵$\\Sigma$的无偏估计**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;由样本协方差矩阵$S$定义：\n",
    "$$\n",
    "S = \\frac{1}{n-1} \\sum_{i=1}^n (x_i - \\bar{x})((x_i - \\bar{x}))'\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;令：\n",
    "$$\n",
    "A = \\sum_{i=1}^n (x_i - \\bar{x})((x_i - \\bar{x}))'\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据无偏估计的定义，证明样本协方差矩阵$S$是总体协方差矩阵$\\Sigma$的无偏估计，即证明样本协方差矩阵$S$的期望等于总体协方差矩阵$\\Sigma$：\n",
    "$$\n",
    "E(S) = E(\\frac{1}{n-1} A) = \\frac{1}{n-1} E(A) = \\Sigma\n",
    "$$\n",
    "即证明：\n",
    "$$\n",
    "E(A) = (n-1) \\Sigma\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;参考《应用多元统计分析 第五版》（王学民，2017年8月第5版）书中第51页的无偏性一节的推导\n",
    "> 依(2.2.14)式\n",
    "> $$\n",
    "V(\\bar{x}) = \\frac{1}{n^2} \\sum_{i=1}^n V(x_i) = \\frac{1}{n^2} \\sum_{i=1}^n \\Sigma = \\frac{1}{n} \\Sigma\n",
    "$$\n",
    "> 于是\n",
    "> $$\n",
    "\\begin{aligned}\n",
    "E(\\hat{\\Sigma}) \n",
    "&= \\frac{1}{n} E\\Big[ \\sum_{i=1}^n (x_i - \\bar{x}) (x_i - \\bar{x})' \\Big] \\\\\n",
    "&= \\frac{1}{n} E\\Big \\{ \\sum_{i=1}^n \\big[ (x_i - \\mu) - (\\bar{x} - \\mu) \\big ] \\big[ (x_i - \\mu ) - (\\bar{x} - \\mu) \\big ]' \\Big \\} \\\\\n",
    "&= \\frac{1}{n} E \\big [ \\sum_{i=1}^n (x_i - \\mu)(x_i - \\mu)' - n(\\bar{x} - u)(\\bar{x} - u)' \\big ] \\\\\n",
    "&= \\frac{1}{n} \\big[ \\sum_{i=1}^n V(x_i) - nV(\\bar{x}) \\big] \\\\\n",
    "&= \\frac{1}{n} \\left(n \\Sigma - n \\cdot \\frac{1}{n} \\Sigma \\right) \\\\\n",
    "&= \\frac{n-1}{n} \\Sigma\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{aligned}\n",
    "E(A) \n",
    "&= E \\Big[ \\sum_{i=1}^n (x_i - \\bar{x}) (x_i-\\bar{x})' \\Big] \\\\\n",
    "&= n E(\\hat{\\Sigma}) \\\\\n",
    "&= n \\cdot \\frac{n-1}{n} \\Sigma \\\\\n",
    "&= (n-1) \\Sigma\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;因此，样本协方差矩阵$S$是总体协方差矩阵$\\Sigma$的无偏估计。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 习题16.3\n",
    "&emsp;&emsp;设$X$为数据规范化样本矩阵，则主成分等价于求解以下最优化问题：\n",
    "$$\n",
    "\\begin{array}{cl}\n",
    "\\min \\limits_L & \\|X - L\\|_F \\\\\n",
    "\\text{s.t.} & \\text{rank}(L) \\leqslant k\n",
    "\\end{array}\n",
    "$$\n",
    "其中，$F$是弗罗贝尼乌斯范数，$k$是主成分个数。试问为什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答思路：**  \n",
    "\n",
    "1. 给出弗罗贝尼乌斯范数的定义\n",
    "2. 给出矩阵的最优近似\n",
    "3. 证明主成分是该最优化问题的最优解，该最优化问题的最优解是$X$的主成分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答步骤：**   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第1步：弗罗贝尼乌斯范数**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第15章的定义15.4的弗罗贝尼乌斯范数：\n",
    "\n",
    "> &emsp;&emsp;**定义15.4（弗罗贝尼乌斯范数）** 设矩阵$A \\in R^{m \\times m}$，$A = [a_{ij}]_{m \\times m}$，定义矩阵$A$的弗罗贝尼乌斯范数为\n",
    "> $$\n",
    "\\|A\\|_F = \\left( \\sum_{i=1}^m \\sum_{j=1}^n (a_{ij})^2 \\right)^{\\frac{1}{2}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第2步：矩阵的最优近似**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据书中第15.3.2节的定理15.3：\n",
    "\n",
    "> &emsp;&emsp;**定理15.3** 设矩阵$A \\in R^{m \\times n}$，矩阵的秩$\\text{rank}(A) = r$，有奇异值分解$A=U \\Sigma V^T$，并设$\\mathcal{M}$为$R^{m \\times n}$中所有秩不超过$k$的矩阵的集合，$0 < k < r$，若秩为$k$的矩阵$X \\in \\mathcal{M}$满足\n",
    "> $$\n",
    "\\big \\| A - X \\big \\|_F = \\min_{S \\in \\mathcal{M}} \\big \\| A - S \\big \\|_F\n",
    "$$\n",
    "> 则\n",
    "> $$\n",
    "\\big \\| A - X \\big \\|_F = (\\sigma_{k+1}^2 +\\sigma_{k+2}^2 + \\cdots + \\sigma_n^2)^{\\frac{1}{2}}\n",
    "$$\n",
    "> 特别地，若$A'=U\\Sigma' V^T$，其中\n",
    "> $$\n",
    "\\Sigma' =\n",
    "\\left [ \\begin{array}{cccccc}\n",
    "\\sigma_1 & & & & & \\\\\n",
    "& \\ddots & & & 0 & \\\\\n",
    "& & \\sigma_{k} & & & \\\\\n",
    "& & & 0 & & \\\\\n",
    "& 0 & & & \\ddots & \\\\\n",
    "& & & & & 0\n",
    "\\end{array} \\right] = \n",
    "\\left[ \\begin{array}{cc} \n",
    "\\Sigma_k & 0 \\\\\n",
    "0 & 0\n",
    "\\end{array} \\right]\n",
    "$$\n",
    "> 则\n",
    "> $$\n",
    "\\big \\| A - A' \\big \\| = (\\sigma_{k+1}^2 +\\sigma_{k+2}^2 + \\cdots + \\sigma_n^2)^{\\frac{1}{2}} = \\min_{S \\in \\mathcal{M}} \\big \\| A - S \\big \\|_F\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第3步：证明命题**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;根据样本矩阵的奇异值分解算法（主成分分析算法），针对$m \\times n$样本矩阵$X$，需要表示为\n",
    "$$\n",
    "X' = \\frac{1}{\\sqrt{n-1}}X^T\n",
    "$$\n",
    "对优化问题的目标函数进行如下变换：\n",
    "$$\n",
    "\\begin{array}{cl}\n",
    "\\min \\limits_L &\\displaystyle \\left \\|\\frac{1}{\\sqrt{n-1}} X^T - \\frac{1}{\\sqrt{n-1}} L^T \\right \\|_F \\\\\n",
    "\\text{s.t.} & \\text{rank}(L) \\leqslant k\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;令\n",
    "$$\n",
    "X' = \\frac{1}{\\sqrt{n-1}}X^T, L' = \\frac{1}{\\sqrt{n-1}}L^T \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;则原最优化问题等价于\n",
    "$$\n",
    "\\begin{array}{cl}\n",
    "\\min \\limits_L & \\|X' - L' \\|_F \\\\\n",
    "\\text{s.t.} & \\text{rank}(L) \\leqslant k \\\\\n",
    "& \\displaystyle  X' = \\frac{1}{\\sqrt{n-1}}X^T \\\\\n",
    "& \\displaystyle  L' = \\frac{1}{\\sqrt{n-1}}L^T \n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;对$X'$进行奇异值分解，根据已知主成分个数为$k$，$X'$的主成分个数也为$k$个，则\n",
    "$$\n",
    "X' = U' \\Sigma' V'^T\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;结合第2步，可知变换后的最优化问题和定理15.3类似，形如：\n",
    "$$\n",
    "A \\sim X' \\\\\n",
    "S \\sim L'\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;可找到 $\\Sigma' = \\text{diag} (\\lambda'_1, \\lambda'_2, \\cdots, \\lambda'_k, 0, 0, \\cdots, 0)$ 使得变换后的目标函数取最小值，即存在矩阵满足主成分条件。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考文献"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "【1】方开泰. 实用多元统计分析[M]. 1989-09:88-89  \n",
    "【2】王学民. 应用多元统计分析 第五版[M]. 2017-08:51  "
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c50c7698ff25eb1d7dce4c52e3d5cc14b3b23d6a97d4fb70aad8c815eba6f63e"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "273.188px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
